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ABSTRACT 


The 21st-century skills needed to face the speed of understanding technology. 
Such as critical thinking in computer vision and robotics literacy, any student 
is hampered by the programming that is considered complicated. This study 
aims at the improvement of student embedded system programming 
competency with computer vision and mobile robotics integration approach. 
This method is proposed to attract the students to learn about embedded system 
programming by delivering integration between computer vision and robotics 
using the P-D controller since both of the fields are closely related. In this 
paper, the researcher described computer vision programming to get the data 
of captured images through the camera stream and then delivered the data into 
an embedded system to make the decision of robot movement. The output of 
this study is the improvement of a student’s ability to make an application to 
integrate a sensor system using a camera and the mobile robot running follow 


the line. The result of the test shows that the integration method between 
computer vision and robotics can improve the student’s programming 
comprehension by 40%. Based on the Feasibility test survey, it can be 
interpreted that from the whole assessment after being converted to qualitative 
data, all aspects of the learning stages of programming application tested with 
the integration of computer vision and robotics fall into the very feasible 
category for used with a percentage of feasibility by 77.44%. 
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1. INTRODUCTION 

Very rapid technological advances affect the speed of understanding technology. This is a challenge 
in the manufacturing sector in the industrial world which must meet consumers’ needs consistently and with 
high quality [1]. To support this, in this modern era, it is very important to understand 21st-century skills which 
include computational thinking [2] as stated in 21st-century learning framework [3]. Based on the literature, 
programming competence strongly supports 21st-century skills. As applied to STT-PLN, to support this, 
STT-PLN has participated to improve student’s comprehension of microcontrollers by creating practical 
modules and obtaining an increase in the value of respondents by 7.8% [4]. 

Robotic courses are often encountered in universities with major electrical engineering. Broadly 
speaking, the automation system in the industrial world is included in the electrical engineering students [5]. 
As in the Universitas Katolik Indonesia Atma jaya, Robotics is taught as one of the subjects with a lot 
of interest. The purpose of this course is to create a robotic-based tool/technology. In general robotic lesson 
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does not only teach the students to become experts in the field of robotics but also to be able to develop essential 
competencies to be succeed in the real world [6]. 

Recently, Robotics has a big impact on the manufacturing industry [7]. Robotics is closely related to 
programming, and currently increasingly associated with the field of computer vision [8]. In the current era, 
Robotics is not far away from the application of artificial intelligence [9] as well in performing fast learning to 
obtain decision options based on multi-input multi-output problems [10]. The control system theory which is 
inseparable from mathematics is also applied to the results of this learning. Basically, there are a lot of 
enthusiasts to learn about robotics, but they are hampered by the programming that is considered complicated 
[11]. As a study conducted at the Department of School of Computer Science, University of Lincoln, UK, 
the method of integration between computer vision and robotics 1s very possible to be taught [12]. 

In addition to programming, the number of sensors that need to be implemented in robots also 
becomes a problem in assembling robots. This is because if there is one sensor that has a problem, it can cause 
a fatal error at the robot's rate. Similar to the line follower robot, it moves based on lines, so that there are more 
than one sensor needed to make the robot see the line below it [13-16]. 

For this reason, this integration method is proposed to encourage the focus of students learning about 
robot programming. In this study, the robot is implemented as a line follower robot equipped with a camera as 
a sensor, Raspberry Pi as an embedded system/processor, IC L293d as a motor driver, and 2 DC 12v motors 
as movers. We use Raspberry Pi as an embedded system that helped in data processing because it could be 
used effectively and smoothly for detection work [17-20]. We replace the photodioda sensor used for line 
follower robots to be a camera because in addition to the use of the cameras, it is easier than making sensor 
modules. Students can learn how to acquire images, and be processed in such a way that they can be data that 
is ready to be processed as a reference for moving motors. To support image processing, the OpenCV 
library is used in this study refers to many projects related to image processing that were also completed with 
OpenCV [21-24]. 


2. RESEARCH METHOD 

Programming comprehension receive special attention and has been applied to the educational 
curriculum in many developed countries. However, programming has not received much attention by 
educational institutions in Indonesia. Although there are already a number of institutions that hold 
programming club activities, it is not enough as we should consider the benefits of programming to train 
student's critical thinking. Efforts are needed to further introduce programming to students. Therefore, this 
study proposed a method in which computer vision and robotics approaches are integrated to improve student’s 
comprehension of programming. The integration steps can be seen in the following Figure 1. 

Figure 1 shows that in order to increase the student's robot programming comprehension, the most 
important thing to do is doing study literature. At this stage, all knowledge about robot programming and 
computer vision is explored. Materials about programming are obtained by using reference books, surfing 
the internet, and presentation slides provided by the instructor. Broadly speaking, things explored include how 
the robot can recognize lines based on image acquisition from the camera, how to process data obtained from 
data acquisition from the camera, and how to use the image acquisition data that has been processed to be able 
to move the robot. 
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After a literature study has been carried out, the researcher started to analyze the needs. The needs 
analysis process is used to find out what hardware is used for image acquisition; embedded systems used to 
process data; hardware to make robots can move based on data that has been processed; and the hardware 
assembly structure of the robot. In addition to analyzing hardware requirements, the software is no less 
important to analyze, such as what software is used to program an embedded system in acquiring images from 
cameras, processing data acquisition, programming languages, and sending data to actuators to move a robot. 
In addition to hardware and software requirements, a control system is also needed to make the robot run stably 
in following the line. Then, the researcher implemented a control system together in data processing to produce 
mature data that is ready to be sent to the robot actuators. 

After all aspects of the needs have been identified, then the process of assembling and programming 
robots begins. There are several choices of programming languages which are suitable for computer vision and 
robotics. One that is used in this study is the Python programming language. 

After the assembly and programming process, the last is evaluation. There are 2 aspects of 
an evaluation carried out in this study. The first is image acquisition, it 1s used to test programs made by 
students for robots to acquire data from images captured by cameras. The second test is the robot running test, 
which is the robot runs on a white track with a predetermined black line. 

To test the level of student's comprehension, the pretest-posttest technique was used in this study. 
There were 22 respondents involved in this study. The pretest was given at the beginning of the material to 
measure the extent of students comprehension of robot programming. After the achievement is done, 
the treatment begins using the method as stated in Figure 2. Finally, after completing the evaluation phase, 
a post-test was given to measure the extent of students comprehension after the learning process. After 
obtaining pretest and posttest data, an analysis was conducted to measure the level of change in students' 
understanding of robot programming with the integration of computer vision and robotics. 

In addition to the Pretest-Posttest to test the student's programming comprehension improvement, data 
analysis techniques in this study were carried out using a quantitative descriptive approach. The quantitative 
descriptive approach was tested using descriptive statistics. The purpose of this analysis 1s to test the Feasibility 
of the integration of computer vision and robotics methods in helping to teach about programming. Quantitative 
data is in the form of calculation and/or measurement figures and can be processed by adding up and comparing 
with the expected number so that a percentage is obtained. So the feasibility percentage is determined by 
the following calculation: 


Percentage (%) = ee ee 100% (1) 


Total Ideal Score 


The analysis of numerical data in this study uses a Likert scale. This is because the Likert scale is 
more reliable than the single team scale. After the percentage scale is formed, the data is interpreted into 
the Score Conversion table as in Table 1. 


Table 1. Score Conversion 


Percentage (%) Criterias 
100 — 76 Very Feasible 
75-51 Feasible 
50 — 26 Not Feasible 

25 —0 Very Infeasible 


3. RESULTS AND ANALYSIS 
3.1. Hardware 

The hardware needed for programming the integration of computer vision and robotics includes 
a camera. The camera is needed as a substitute for sensors to detect lines by acquiring path images to be 
represented in pixel coordinates in programming. The camera used in this study is Pi Camera V2 which can be 
used to take pictures with a resolution of 1080p and 720p, while for video can be recorded with a resolution 
of 640x480p. The embedded system device as a data processor used in this study is the Rspberry Pi 3 B + 
model that uses ARM (Advanced RISC Machine) as a processor and 1 GB of RAM. In addition to cameras 
and embedded systems, the robot is equipped with a motor driver module with IC L293d which is used to assist 
in regulating the speed and direction of rotation of two DC12v 450rpm motors on the robot. Then, to assemble 
all hardware components, a chassis made of acrylic boards with a thickness of 3mm is used in this study. 
In addition to the hardware from the robot itself, a PC/Laptop is needed to help the programming process by 
means of the Raspberry Pi remote system to the PC/Laptop. 
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3.2. Software 

The softwares used in this study include the Operating System used on Raspberry Pi 3 B + models, 
applications for Raspberry Pi remotes using a PC using a WIFI connection, applications for making code, and 
libraries needed to program robot hardware. The operating system used in this study is Raspbian Stretch. 
To support the robot programming process, the VNC application is used to operate Raspberry Pi remotely 
through a laptop. Furthermore, the application used to create code on the Raspberry Pi is the default text editor 
from Raspbian. The last but not the least software 1s OpenCV library that was used to assist in making code to 
acquire image data captured by cameras. 


3.3. Programming 


In this phase, Python is used as a programming language to make robots function as line follower 
robots. There are 2 phases of programming carried out on this integration method. 


3.3.1. Image data acquisition 

Image data acquisition is used to process images captured by the camera and take data every pixel 
in the image. The purpose of this process is to produce the midpoint of the detected object, in this case 
a black line in the robot's path. The results of the programming process in image data acquisition can be seen 
in Figure 2. 


Captured Image Grayscale Thresholding 


me” oe” 
; 


Object Center Coordinate Edge Detection 


Figure 2. Image data acquisition 


3.3.2. Control system and data delivery to actuators 

After the center coordinate data of the object captured by the camera is obtained, the data is then used 
to be processed again using a control system to be a value to control the rotation, direction, and speed of 
the two actuators. The control system used in this study is the P-D controller. The stages of implementing data 
from the center object reading into the control system are as follows. The frame divided into 9 parts, each part 
has a weight value with the following details stated in Table 2. Which the weight value will be considered as 
an error value of the X value in the middle of the frame. 


Table 2. The weight value determination of each frame 
The Range of X (px) Weigth Value 
0 <x< 20 
20<x<37,5 
37,5<x<55 
55<x<72,5 
72,5<x<87,5 
87,5<x<105 
105<x<122,5 
122,5<x<140 
140<x<160 


Loa Sey ek 


The next stage is the application of the weight value obtained from image processing into the control 
system. This study applies PD controller to support the control system. There are many studies that apply PD 
controllers to robots [25-26] and also develop PD controllers so that they are more robust [27]. The PD 
controller system parameter is based on the diagram which can be seen in Figure 3. 
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Based on the diagram block in Figure 3, an algorithm can be interpreted to regulate robot motion into pseudo 
code as follows: 


setpoint = 0 

previous_error =O 

Kp = (tuning) 

Kd = (tuning) 

error = setpoint — weight_value 
derivative = (error — previous_error)/dt 
pd_out = (Kp * error) + (Kd * derivative) 
previous_error = error 

wait(dt) 


PWMLeft(constant_speed + pd_out) 
PW MkRight(constant_speed-pd_out) 


By using the algorithm above, the robot’s logic can be interpreted as shown in flowchart Figure 4. 
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Figure 3. P-D controller diagram block Figure 4. Flowchart system 


3.4. Evaluation 
3.4.1. Robot testing result 
The results of computer vision and robotics programming integration testing can be seen in Figure 5. 





Figure 5. Mobile robot running test result 
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3.4.2. The result of pretest posttest 

The results of student programming comprehension improvement are assessed based on the pretest 
and posttest questions done by students at the beginning and the end of the meeting. Pretest and posttest 
questions were done by students amounted to 20 items including material about Image Processing, Python 
programming, use of OpenCV, P-D Control System, and GPIO Programming on Raspberry Pi 3. Pretest and 
posttest questions were done for 50 minutes with the expectation of maximum results. Based on the data 
obtained, the average value of the pretest and posttest of all students has increased. An improvement in 
the average pretest and posttest scores can be seen in Figure 6. 
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Figure 6. The improvement of average pre-test and post-test result 


Student programming comprehension results improved by 40% from average Pre-Test score of 51.82 
to average Post-Test score of 72.55. Based on this result, the learning methods of Computer Vision and 
Robotics integration are appropriate to be applied to the Robotics course. 


3.4.3. Feasibility test result 

The Feasibility test in the study is based on the usability aspect of applying the learning stages of 
programming with the integration of computer vision and robotics. Responses to usability aspect were collected 
using a questionnaire with 7 items representing sub-aspects of Understandability, Operability, and Learnability. 
Data on the results of the Feasibility test in this study can be seen in Table 3. 


Table 3. Feasibility test questionnaire results 


Aspect Sub-aspects Indicators Avg. Point Max. Avg. Point 
ee 1 3.05 4 
Understandability 7 397 4 
2 3 3.14 4 
5 23 4 2.95 4 
fae} 
E Operability 5 3 97 4 
6 2.82 4 
Learnability 7 3.18 4 
Total 21.68 28 


Table 3 shows that the total of average feasibility point is 21.68 instead of 28. According to the data, 
the feasibility percentage can be shown using this equation as stated before. The result of feasibility percentage 
calculation can be seen as follows: 


Total Avg.Point 


Feasibility = Taa Ag PONA x 100% (2) 
Feasibility = — x 100% (3) 
Feasibility = 77.44% (4) 
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Based on the Feasibility test survey, it can be interpreted that from the overall assessment after being 
converted to Qualitative Data, all aspects of the testing of the application of learning programming stages with 
the integration of computer vision and robotics are included in the Very Feasible category for use with 
a percentage of feasibility by 77.44%. 


4. CONCLUSION 

The impact of applying the learning stages of programming with the integration of computer vision 
and robotics can improve student’s programming comprehension in robotics. Student’s comprehension 
improvement was observed based on pre-test and post-test results. The average value of student’s 
understanding increased by 40% from the original average pre-test score was 51.82 to 72.55. In addition, based 
on the Feasibility test that has been analyzed, this study falls into the very Feasibility category for use with 
a percentage of Feasibility of 77.44%. The integration of Computer Vision and Robotic programming can still 
be developed for further research such as the introduction of objects on the camera, calculation of the object 
distance from the robot, and the object tracking system on the robot. 
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